Here are the five models that we used, in increasing order of adversarialrobustness: = 0,0.5,1.0,3.0,5.0. Three ImageNet-trained vision transformer (ViT) models [47] were obtained from pytorch-image-models [48]. Note that the "imagenet1k" suffixinthe model names does not mean the model wasonly trained on ImageNet1K. Observation: A vision transformer (ViT-S) indeed shows higher error consistency with ResNet-50 than with BagNet-9 (see Table 1). Further insights could be gained by testing successively more constrained versions of the samebasemodel.

Toanswer this question, we tested human observers on abroad range of out-ofdistribution (OOD) datasets, recording 85,120 psychophysical trials across 90 participants.

A central problem in cognitive science and behavioural neuroscience as well as in machine learning and artificial intelligence research is to ascertain whether two or more decision makers---be they brains or algorithms---use the same strategy. Accuracy alone cannot distinguish between strategies: two systems may achieve similar accuracy with very different strategies. The need to differentiate beyond accuracy is particularly pressing if two systems are at or near ceiling performance, like Convolutional Neural Networks (CNNs) and humans on visual object recognition. Here we introduce trial-by-trial error consistency, a quantitative analysis for measuring whether two decision making systems systematically make errors on the same inputs. Making consistent errors on a trial-by-trial basis is a necessary condition if we want to ascertain similar processing strategies between decision makers. Our analysis is applicable to compare algorithms with algorithms, humans with humans, and algorithms with humans. When applying error consistency to visual object recognition we obtain three main findings: (1.) Irrespective of architecture, CNNs are remarkably consistent with one another.

CPDTrack exhibits strong generalizability across various challenging benchmarks.

Stimuli are visualized in Figures 7 and 8. Our Python library,"modelvshuman", to test and benchmark models against high-quality human These two models are referred to as ViT -L (14M) and ViT -B (14M) in the paper. Note that the "im-agenet1k" suffix in the model names does not mean the model was only trained on ImageNet1K. We then made two predictions which we test here. While this relationship is not perfect (e.g., the difference is small for silhouette Prior to the experiment, visual acuity was measured with a Snellen chart to ensure normal or corrected to normal vision. Our experiment was a standard perceptual experiment, for which no IRB approval was required.

The supplementary material is structured as follows. We start with terminology in Section S.1, afterwards we In addition to method details, we provide extended experimental results in Figure SF.3 (error consistency of all Furthermore, Figure SF.4 visualises qualitative error differences by plotting which stimuli were particularly easy We would like to briefly clarify the name error consistency . Two decision makers necessarily show some degree of consistency due to chance agreement. How much observed consistency can we expect at most for a given expected consistency? We distinguish between two cases.